Expand Glyphs multiple languages syntax when building UFOs - #1162

So.. I’m not convinced a physical line is the right unit for this transformation. FEA defines line endings as ordinary whitespace therefore valid statements may span lines or share one.

Unlike with the existing variable_features.py, expanding multiple languages cannot operate on locally delimited syntax because it needs to understand the type and scope of the following FEA statements, which means reconstructing statement boundaries, nesting, definitions and lookup references, all through a bunch of fragile regexes...

E.g. this would be valid Glyphs syntax that the current code leaves unexpanded because another statement follows on the same line:

language AZE CRT; sub i by idotaccent;

Or take this one:

language AZE CRT;
lookup idot { sub i by idotaccent; } idot;
sub Scedilla by Scommaaccent;

here the scanner can't detect a lookup's opening and closing braces that occur on the same line (brace depth is 0) and treats the lookup as continuing to the end of the language section, so CRT only receives lookup idot; while the inline sub rule gets silently ignored.

There are more of these all stemming from conflating line with statements. E.g. @Ced = [Scedilla]; sub @Ced by Scommaaccent; would be classified as one definition and the sub not replayed. Or line break after lookup or markClass would prevent the definition from being recognized.

Fixing these (or even just rejecting them with a loud error to avoid silent wrong output) risks turning this into a partial second FEA parser inside glyphsLib, needing knowledge of lookup, class/markClass semantics. Besides, this operates on incomplete feature text before includes gets resolved, so there's that too.

So I’m not yet convinced glyphsLib can transform the extension safely without duplicating parts of feaLib. And maybe feaLib would not necessarily be the wrong layer, despite not being official FEA... I honestly don't know, sorry for being inconclusive.

(and then there is the question of also needing to implement this in fontc/fea-rs which just hit 1.0 this week.. at some point I would rather stop introducing new features in both compilers)

You're right, and the line-based scanning isn't salvageable. Thanks a lot for writing the counterexamples out — all four reproduced exactly as you described, including the two that fail silently.

I've rewritten the expansion on feaLib's own Lexer instead of on the text:

The part I'd gently push back on is the "partial second FEA parser" worry — tokenizing is precisely what removes the need for one. Lexer already gives me statement boundaries (a ; at brace depth zero, wherever it sits on the page), and it already tells a brace inside a comment or a string from a real one. What's left is classifying the head of each statement, and that grammar is five cases:

• lookup NAME [useExtension] { → definition: emitted once, replayed as lookup NAME;
• @Class = … → definition: emitted once
• markClass … → definition: emitted once
• include(…) → bail out (below)
• anything else, lookup NAME; references included → repeated verbatim

with the scope ending at the next language/script or an unmatched }. No glyph classes to resolve, no lookup bodies to understand, no name resolution. The emitted text is sliced out of the original by offset, so comments, indentation and blank lines survive untouched.

TokenExpander in builder/tokens.py already runs inside _to_ufo_features and rewrites Glyphs-only $[…] / ${…} / $name syntax into plain FEA — with a hand-rolled predicate subparser and an eval() for the arithmetic. In my opinion, it is more Glyphs-specific machinery than the five heads above. I know those tokens are locally delimited and the shorthand isn't — that's a fair distinction. My point is only that lexing moves that line: on a token stream a statement boundary is as local as the } that closes a ${.

On includes: agreed, they can't be replayed, so a multi-tag scope containing one is left alone entirely. The shorthand is invalid FEA, so feaLib then reports it — the failure stays loud instead of becoming a silently partial expansion.

What this does not address is your second comment about fontc parity, because it still fails unless the same expansion is implemented there too. The portability argument in my PR description ("the UFO gets plain FEA, so any compiler reads it") doesn't cover fontc, and I shouldn't have framed it that broadly. The parity cost you're describing is real and this patch doesn't remove it.

One thing that might change the calculus: Glyphs 4 emits it from automatic feature generation. I took two of our Glyphs 3 sources (appVersion 3517 and 3526), opened and saved each in Glyphs 4 (appVersion 4004), and in both the automatic locl came back rewritten the same way:

# before, appVersion 3517
language AZE;
lookup locl_latn_0 {
    sub i by idotaccent;
} locl_latn_0;
language CRT;
lookup locl_latn_0;
language KAZ;
lookup locl_latn_0;
...

# after, appVersion 4004
language AZE CRT KAZ TAT TRK;
lookup idotaccent {
    sub i by idotaccent;
} idotaccent;

No edits by the designer, the feature still flagged automatic before and after, and both fonts went from ten single-tag statements to two single plus the same two shorthand ones (AZE CRT KAZ TAT TRK and ROM MOL). So this isn't an opt-in extension that some sources happen to use — it's ordinary Glyphs 4 output, and a source that built yesterday stops building after a save, in feaLib and fea-rs alike. Two fonts against one Glyphs 4 build is still not a survey, but it reproduces rather than looking like a fluke.

As I read it, there are three ways this could go:

1. Token-based expansion as described — I can push it to this branch.
2. Detect and reject, no expansion. A language statement with more than one tag becomes a clear error instead of Expected ';' pointing at column 14. Needs no statement semantics at all, needs no fontc counterpart, and is strictly better than today.
3. Drop it — it belongs in feaLib/fea-rs, or nowhere.

I can only guess which one you'd prefer. But if Glyphs emits the shorthand on save, everything that reads .glyphs sources ends up having to solve it, and for the UFO path this is the one place to do it once — fontc would still need its own, which is the cost you flagged, though I don't think that argues for nobody doing it.

Happy to push (1) here, or cut it back to (2) — whichever you'd merge. For what it's worth on the approach itself: I ran the tokenizer over ~1100 feature code blocks from 35 production .glyphs files and all of them lex cleanly on the raw pre-include text, which is what convinced me it holds up outside the test suite.

Happy to push (1) here

thanks, since you have done it already and sounds like it's better than the line-base scanner, you may as well push it

Glyphs 4 emits it from automatic feature generation

oh, great. That means we can't dismiss this and will have to support it in fontc anyway (googlefonts/fontc#2126). And maybe in feaLib as well, which would make the current PR redundant.

We documented our fea additions. There is a branch for each topic: https://github.com/schriftgestalt/feature_file_workshops (This is not meant as a request to add it to the spec as is, just to have written it down properly.)

Thanks Georg, I briefly checked Adobe's upstream repo the other day but couldn't find your multi-lang extension proposal. I found it now here: adobe-type-tools/feature_file_workshops#8

Pushed. The line scanner is gone — multi_language.py now walks feaLib's own Lexer, so a statement ends at the first ; at brace depth zero wherever it sits on the page, and a brace inside a comment or a string is never code. Your four counterexamples are in the tests under their own names:

• test_statement_sharing_the_line_with_the_shorthand_is_expanded
• test_lookup_closed_on_one_line_does_not_swallow_what_follows
• test_definition_sharing_a_line_with_a_rule_replays_the_rule
• test_line_break_after_lookup_keyword_is_still_a_definition

Georg — thanks, adobe-type-tools/feature_file_workshops#8 is exactly the grammar this implements (language <tag>+ [exclude_dflt|include_dflt] [required];, following rules registered under every listed tag for the current script), so nothing to change here. And if the conclusion is that this belongs in feaLib rather than at the UFO boundary, that makes sense to me — expanding here is only worth it for as long as the compilers don't.

Thanks.

adobe-type-tools/feature_file_workshops#8 is exactly the grammar this implements

What does Glyphs 4 do with dflt inside a multi-tag language statement? The proposal says it must be specified alone (https://github.com/schriftgestalt/feature_file_workshops/blob/2d1d7a058e2642a9bf6ef0f6dd8594ea7d7d40e8/OpenTypeFeatureFileSpecification.md?plain=1#L1219), and Glyphs 3.4.1 refuses to export it.

E.g. with the following feature prefix:

languagesystem DFLT dflt;
languagesystem latn dflt;
languagesystem latn AZE;

and this locl feature:

script latn;
language dflt AZE;
sub i by idotaccent;

Glyphs 3.4.1 says Error: Regular: Expected ";" after language statement, found in feature "locl", line 2.

Your PR expands it anyway, to language dflt; + the original rules and then language AZE; + the replayed rules.
But since language AZE; implies include_dflt, AZE gets the substitution twice, once inherited from dflt and once replayed, as two distinct lookups with identical content both registered under AZE. This is harmless for shaping but superfluous.

Could you check what Glyphs 4 does with it? If it rejects it too, I'd leave such a statement untouched and let feaLib report it, like the other shapes you don't classify. If it accepts it, knowing what lands in the GSUB would settle what the expansion should emit.

Glyphs 4.0.1 (4004) rejects it too, with the same error as 3.4.1 — this is the compiler in the Features tab:

script latn;
language dflt AZE;
sub i by idotaccent;

→ Expected ";" after language statement on line 2, locl marked as failing.

So a multi-tag statement listing dflt is now left untouched, with a warning saying why, and feaLib reports it as before. Only dflt is guarded, not DFLT — expanding a DFLT statement still fails loudly in feaLib, while expanding dflt compiled and produced the redundant duplicate you described.

Looking for the same class of problem elsewhere turned up three more — a lone \r (feaLib ends a line there too, so the offsets went out of step and the replay silently lost its rules), a missing semicolon, and a shorthand outside any block — all now left alone, with two invariants over a corpus of awkward sources in place of one example each. Three existing tests wrap their snippet in a feature block now, since the entry point only ever sees the full assembled text, where a top-level language is invalid.